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Abstract  A 

v  A  If 

Each  iteiater  generated  by  the  Generalized  Conjugate  Gradient  Method  >f  Concns 
and  Golub  p]-and  Widlun<r-{3)'  is  shown  to  be  the  best  approximation  U  >  the 
solution  from  a  certain  affine  subspace  (although  not  from  the  ’natural*  affine 
Krylov  subspace).  This  property  is  used  to  improve  the  error  bounds  given  by 


A  Note  on  the  Generalised 
Conjugate  Gradient  Method 

Stanley  C.  Ei  sens  tat 

Research  Report  #228 
March  1982 


This  research  was  supported  in  part  byjGNR  Grant  N00014-76-C-0277  and  NSF  Grant 
MCS-8W4874. 


1 


1.  Introduction 

The  Generalized  Conjugate  Gradient  Method  of  Concus  and  Golub  [l]  and  Widlund  [3]  is 
an  iterative  method  for  solving  a  system  of  linear  equations  Ax  ™  b  when  the  coefficient  matrix 
A  is  real  and  has  positive  definite  symmetric  part  M  *  (A+Al)/2: 

Let  *<•>  BE  GIVEN  AND  SET  x(~lj  —  0. 

FOR  m  —  0  STEP  1  UNTIL  “CONVERGENCE"  DO 
Solve  Mi/m)  —  i  -  Ax(m) 

COMPUTE1  Pm  ■=  t/m)) 

IF  m  —  0  THEN 

SET  "«+i  “  1 

ELSE 

COMPUTE  Wm+1  -  [1  + 

Compute  x(m+1)  «*  +  wm+l  (t/m)  +  x(m)  — 

Let  A  **  M—N,  whence  —N  *“  (A— A1)/ 2  is  the  skew-symmetric  part  of  A,  and  let 
K  **  M~lN.  Then  it  can  be  shown  that  the  iterate  lies  in  the  affine  Krylov  subspace 

z(#)  +  Span{t/#),  Kv{*\  icV0),  ....  Km~1v{(i)}  m  x(#)+5m 
and  is  characterized  by  the  Galerkin  condition 

(r,  Ae^)  ■*  0  for  all  t  €  Sm,  (1.1) 

where  m  x^—x  (see  [3]).  Moreover, 

xM-z  +  Pm(K) ew  (1.2) 

where  pm(p)  is  an  even  (odd)  polynomial  of  degree  at  most  m  for  m  even  (odd)  and  pm(l)  “  1 
(see  [3]). 

In  this  paper,  we  show  that  is  the  best  approximation  to  x  from  a  certain 
m-dimensional  affine  subspace  (but  not  from  the  affine  Krylov  subspace  £„,)  and  use  this 
property  to  improve  the  error  bounds  given  by  Widlund  [3]  and  Hageman,  Luk,  and  Young  [2]. 


1  (y,z)  denotes  the  Euclidean  inner-product. 


Notation:  (y,z)M  denotes  the  M-inner  product  (My,z)  and  denotes  the 
corresponding  norm.  Note  that 

(Ky,z)u  —  ( Ny,z )  —  ~(y,Nz)  —  -( My,M~lNz )  —  ~(y,Kz)M 
so  that  K  is  skew-symmetric  with  respect  to  and  (Kz,z)u  -*  0  for  all  z. 

2.  An  Alternative  Characterisation 

In  this  section,  we  show  that  the  iterate  generated  by  the  Generalised  Conjugate 
Gradient  Method  is  the  best  approximation  to  z  with  respect  to  a  certain  m-dimensional  affine 
subspace,  but  not  with  respect  to  the  affine  Krylov  subspace  z^+Sm  (unless  z^  ■*  z).  The 
cases  m  even  (■=  2Jt)  and  m  odd  (=  2Jt+l)  are  treated  separately. 

Theorem  2.1:  z^  €  z^+(/+K)$j4  and 

(z,  x{tk]~x)M  -  0  for  all  z  €  (/+K)SU, 

whence 


l|z(l*)-*||w  —  min  {h-*\\u  I  V  €  x{9]+{I+K)Su}  . 

Proof: 

Since  pu(— 1)  ■*  Pu(l)  **  1  (recall  that  pu  is  even),  p,*(p)  can  be  written  in  the  form 
P**(P)  —  1  +  (I+p)  »**-*(#») 

where  ffj*_j(p)  is  a  polynomial  of  degree  at  most  21—2.  Therefore,  by  (1.2), 

z(**>  -  z  +  e,#)  +  {I+K)nu_t(K)(I-K)e{9) 

-  x<*>  -  ( I+K ) 

€  z,#)+(/+A)SJjk  . 

If  z  6  (I+K)Stk,  then  z  —  (I+K)v  for  some  u  6  and 

(z,  x{U)~z)y  -  (A4(/+A>,  e{U))  -  (u,  Ae(u)  -  0 

□ 


by  the  Galerkin  condition  (1.1). 


However,  is  not  the  best  approximation  to  x  from  x^'4  Su.  To  see  this,  note  that 

*(M)-*)*  -  -  «I-K)<(9\ 

- (*<**),  eW)M  +  (eW-eC\  eW)M  +  (#*<*>,  pu(K)e(%  . 

By  Theorem  2.1,  e^—e®  »»  *(**)_*(•)  g  (/+K)SU  and  the  second  term  vanishes.  Since  K  is 
skew-symmetric  with  respect  to  (-,-)w  and  is  even,  the  third  term  also  vanishes.  Therefore, 
t/#>  €  Sj*  but 

(«<•>.  x'“»-r)u  -  -  (.<«>,  «!“>)„  /  0  , 
unless  x<**>  **  x. 

Theorem  2.2:  z(**+,)  €  x{0)+v{a)+(I+K)$u+l  and 

(2,  z(J*+,)-x)M  -  0  for  all  2  €  (I+K)SU^, 


whence 


Proof: 


{llv-x|)vIv€xW+t/#4(/+K)S,*+1}  . 


Since  Pu+l(l)  —  1  and  PM+1(-1)  —  -Pj*+iU)  “  “1  (recall  that  pu+l  is  odd),  pw+1(/»)  can 
be  written  in  the  form 

Pi4+1(M)  *  p  +  (1+p)  z,*_j(p)(l-p) 

where  wu_,(p)  is  an  odd  polynomial  of  degree  at  most  2k—  1.  Therefore,  by  (1.2), 

z(**+,)  -  x  +  JCe(#)  +  (/+tf)  *U_SK)  {I~K)c{t) 

-  *W  -  (I-K)cW  -  (7+/C)  zIjk_1(/Ot/0) 

—  *«  +  t/#)  -  (/+#) 

6  z('4i/0)+(  I+K)  S,A+1  . 

If  2  €  (Z+ZOSj^t.  then  t  ■*  ( I+K)u  for  some  u  €  SJt+I  and  ’  -•  r  ‘ 

*  ,*  •  ■'  ‘  -  *■ 

(2,  z,w+,)-x)v  -  (M(I+K)u,  e(U+1))  -  (u,^»*+‘))  -  0 
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by  the  Galerkin  condition  (1.1).  □ 

Again,  ar***+^  is  not  the  best  approximation  to  x  from  To  see  this,  note  that 

-  (<W.  PUJK^)U  . 

By  Theorem  2.2,  e^u+^—Ke^  —  *(**+,)_sc(«)_w(«)  g  ( I+K)SU  and  the  second  term  vanishes. 
Since  K  is  skew-symmetric  with  respect  to  and  Psi+l  is  odd,  the  third  term  also  vanishes. 
Therefore,  6  SJJk+I  but 

(VW  x(»+D_x)u  _  («<**+»>,  e(»+t))u  ^  o  , 
unless  x(M+,)  —  x. 

3.  Error  Bounds 

In  this  section,  we  use  the  best  approximation  property  of  the  iterates  {x^}  to  prove  error 
bounds  for  the  Generalized  Conjugate  Gradient  Method. 

Theorem  3.1: 

||x(m)-x||JU,  <  ||9nI(tfX*W-*)||J(/ 

for  any  real  polynomial  qm(ft)  of  degree  at  most  m  satisfying  ym(l)  “  1  and  9m(— 1)  =  (— l)m. 
Proof: 

Let  p  m  x  +  qm(K)J*\  Then  it  can  be  shown  that  y  €  z^+(I+K)Sm  if  m  is  even  (see  the 
first  part  of  the  proof  of  Theorem  2.1)  and  that  y  £  x^-H*^+(  f+K)$m  if  m  is  odd  (see  the  first 
part  of  the  proof  of  Theorem  2.2).  Therefore,  using  either  Theorem  2.1  or  Theorem  2.2, 

□ 

Let  o{K)  denote  the  spectrum  of  K.  Since  K  is  skew-symmetric  with  respect  to  (v)*#.  it 
can  be  shown  that 


.kfr *  , 
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Re  M-0,  |/i»H  <  \\K\\umA 

for  any  fi  £  ff(AT),  and  that 

for  any  real  polynomial  gm(p). 

Corollary  3.2: 

lk(M)-xBw  < 


*ur  +  Mur- 

where  R{A)  —  vl-1  +  n/>1_*+1  . 

Proofs 

Let  —  Tm(iA~lit)/Tm(iA~l)  where  Tm(s)  is  the  mlk  Chebyshev  polynomial.  Since 
Tm(z)  is  even  (odd)  when  m  is  even  (odd),  qm(p)  is  a  real  polynomial  which  satisfies  the  conditions 
of  Theorem  3.1  so  that 

<  lkm(*X*(0)-*)llw  <  l|qm(/f)Uu 

But 


max 


i 


irjivi-1)!  ~  \TJiA")\ 


since  —1  <  iA~ln  <  +1  for  all  ji  £  o{K)  and  |T^(a)J  <  1  for  —1  <  r  <  +1.  Moreover,  it  can 
be  shown  that 


TJiA-')  -  [flM)m  +  l-fifA)]-*]  . 


Therefore,  since  R{A)  >  1, 


m-  +  mu* 


= i*"’- 


□ 


Hageman,  Luk,  and  Young  (2]  proved  Corollary  3.2  for  m  even  by  observing  that  the  even 
iterates  can  also  be  generated  by  applying  conjugate  gradient  acceleration  to  a  certain 


.M 


symmetriiable  ‘‘double”  method.  Widlund  [3]  proved  somewhat  weaker  bounds  for  general  m 
using  a  standard  argument  for  Galerkin  methods. 

The  best  approximation  property  and  the  nesting  of  the  subspaces  { Sm)  guarantees  that 
{||e(*%}  wd  {||e<**+%}  are  both  monotone  decreasing.  Widlund  [3]  gives  a  direct  proof.  The 
following  result  shows  that  both  sequences  must  converge  at  the  same  rate,  contradicting  the 
experimental  results  reported  in  (3]. 

Corollary  3.3: 

A~l  flx(",+l)-*||J,  <  ||x(m)-xflv  <  A  ||x<"-1>-x||„  for  all  m  >  1. 

Proofs 

It  suffices  to  prove  the  right-hand  inequality.  Since  qm(p)  /ipm_1(p)  satisfies  the 
conditions  of  Theorem  3.1, 

l|x(m)-x||v  <  lkm(^Xx(#,-*)llw 
-  A  ||*(",-,)-x|]w  . 

□ 
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